<html>
<head><meta charset="utf-8"><title>refcell-vs-lock · t-compiler/wg-parallel-rustc · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/index.html">t-compiler/wg-parallel-rustc</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html">refcell-vs-lock</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="169889842"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169889842" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169889842">(Jul 08 2019 at 17:47)</a>:</h4>
<p>hey <span class="user-mention" data-user-id="116466">@Zoxc</span>! <span class="user-mention" data-user-id="116009">@nikomatsakis</span> asked me to get up to speed on the (prolific!) work you've been doing on parallelizing rustc, so that i can start reviewing your current PRs, and help write more docs etc</p>



<a name="169889850"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169889850" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169889850">(Jul 08 2019 at 17:47)</a>:</h4>
<p>i did have a few questions though</p>



<a name="169889939"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169889939" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169889939">(Jul 08 2019 at 17:48)</a>:</h4>
<p>in particular, i was wondering whether you put much thought into atomicity and lock order concerns -- i was noticing that a lot of key structures have relatively "fine-grained" locks, in the sense of having lots of fields with individual locks around them</p>



<a name="169889969"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169889969" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169889969">(Jul 08 2019 at 17:49)</a>:</h4>
<p>i was thinking that, compared to when  these were <code>RefCell</code>s, there could be a couple of new problems to worry about:</p>



<a name="169890079"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169890079" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169890079">(Jul 08 2019 at 17:50)</a>:</h4>
<ul>
<li>Atomicity invariants: a  lot of the code tends to hold a field lock for a relatively  short time, but it may  operate on a number of fields in sequence. in sequential  mode, we would be making sure invariants hold on exit from the whole function. but i'm worried that, in concurrent mode, we'll have broken invariants in the middle of a function execution</li>
</ul>



<a name="169890123"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169890123" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169890123">(Jul 08 2019 at 17:51)</a>:</h4>
<p>it's been a bit hard to assess this, though, because the <code>Lock</code> API closely matches the <code>RefCell</code> API, so the diffs aren't showing all the places that reading or mutation occurs</p>



<a name="169890195"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169890195" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169890195">(Jul 08 2019 at 17:51)</a>:</h4>
<ul>
<li>Lock ordering: we occasionally lock while holding another lock, and i wasn't able to find much documentation about lock ordering constraints. if different code locks the same pair of locks in  opposite order, we could get deadlock</li>
</ul>



<a name="169890281"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169890281" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169890281">(Jul 08 2019 at 17:52)</a>:</h4>
<p>i'm less worried about the latter, because it's easier to audit for, though even in  cases  that are "fine" right now it seems helpful to have a documented order to avoid deadlocks creeping in</p>



<a name="169890339"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169890339" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169890339">(Jul 08 2019 at 17:53)</a>:</h4>
<p>overall this makes me a bit worried that we may hit heisenbugs and other hard-to-track-down issues</p>



<a name="169890359"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169890359" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169890359">(Jul 08 2019 at 17:53)</a>:</h4>
<p>(I moved this conversation to its own topic)</p>



<a name="169890366"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169890366" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169890366">(Jul 08 2019 at 17:53)</a>:</h4>
<p>thanks <span aria-label="heart" class="emoji emoji-2764" role="img" title="heart">:heart:</span></p>



<a name="169890503"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169890503" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169890503">(Jul 08 2019 at 17:55)</a>:</h4>
<p>so, anyway, was wondering what your thinking was on these topics. my own inclination would be to use a combination of audit/documentation (which i've gotten a start on), and perhaps refactoring to reduce the need for locks in the first place (as I've been reading through PR history, there's definitely some low-hanging fruit  there). i'm more than happy to take on that work, fwiw</p>



<a name="169892963"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169892963" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169892963">(Jul 08 2019 at 18:26)</a>:</h4>
<p>Changing the <code>borrow_mut</code>/<code>borrow</code> calls to <code>lock</code> would probably be a good idea now, so things look like lock accesses.</p>



<a name="169894608"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169894608" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169894608">(Jul 08 2019 at 18:48)</a>:</h4>
<p>I did look for these issues when I converted the code to using locks (looking at all the accesses for each lock). Documentation won't help much besides possibly prevent regressions here. Testing with <code>rr</code> in chaos mode or with many core CPU might be useful, but testing realistic incremental workloads is tricky. We'd probably need to record actual user workflows for that.</p>



<a name="169897637"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/169897637" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#169897637">(Jul 08 2019 at 19:31)</a>:</h4>
<p>I'm confusing by what you mean by "documentation won't help much" -- like, I don't know what documentation you are referring to, or what you mean by help :)</p>



<a name="170411789"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/170411789" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#170411789">(Jul 08 2019 at 23:49)</a>:</h4>
<p>Hey <span class="user-mention" data-user-id="116010">@Aaron Turon</span> I was talking to <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> and he'd be interested I think in helping out with refactoring things</p>



<a name="170412176"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/170412176" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#170412176">(Jul 08 2019 at 23:56)</a>:</h4>
<p><span class="user-mention" data-user-id="116010">@Aaron Turon</span> <span class="user-mention" data-user-id="116009">@nikomatsakis</span> definitely</p>



<a name="170473717"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/170473717" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#170473717">(Jul 09 2019 at 17:36)</a>:</h4>
<p>fantastic! and yeah, it seems like the best of all worlds is pushing to eliminate (or simplify) shared state wherever we can</p>



<a name="170473876"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/170473876" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#170473876">(Jul 09 2019 at 17:38)</a>:</h4>
<p>i'll work toward some kind of checklist we can parallelize</p>



<a name="170474510"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/170474510" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#170474510">(Jul 09 2019 at 17:45)</a>:</h4>
<p><span class="user-mention" data-user-id="116010">@Aaron Turon</span> <span aria-label="wave" class="emoji emoji-1f44b" role="img" title="wave">:wave:</span> </p>
<hr>
<p>I think one thing that's missing from the data structures / sync code in the compiler is comments on the unsafe code; would be good to change that as part of documenting</p>



<a name="170475542"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/170475542" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#170475542">(Jul 09 2019 at 17:56)</a>:</h4>
<p>I wouldn't bother with larger refactorings for code which does not yet use the query system, unless they can be brought into it</p>



<a name="170475794"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/170475794" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Turon <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#170475794">(Jul 09 2019 at 17:58)</a>:</h4>
<blockquote>
<p><span class="user-mention silent" data-user-id="116010">Aaron Turon</span> <span aria-label="wave" class="emoji emoji-1f44b" role="img" title="wave">:wave:</span> </p>
<hr>
<p>I think one thing that's missing from the data structures / sync code in the compiler is comments on the unsafe code; would be good to change that as part of documenting</p>
</blockquote>
<p>yeah --  in general this seems like a good excuse to make some improvements not even related to parallelism</p>



<a name="170556985"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock/near/170556985" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/refcell-vs-lock.html#170556985">(Jul 10 2019 at 16:36)</a>:</h4>
<blockquote>
<p>i'll work toward some kind of checklist we can parallelize</p>
</blockquote>
<p><span class="user-mention" data-user-id="116010">@Aaron Turon</span> yeah, please let me know and we can start talking about things to do :)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>